fix: prevent evaluation run from getting stuck in Running state when interrupted#449
Open
octo-patch wants to merge 1 commit into
Open
Conversation
…Running state (fixes MadcowD#436) Previously, if an evaluation run was interrupted (e.g. KeyboardInterrupt) or raised an unexpected exception, write_evaluation_run_end was never called, leaving the run permanently stuck in Running state in the store. Add a BaseException handler that sets success=False, records the error message, and calls write_evaluation_run_end before re-raising, so the run is always finalized regardless of how it exits. Also guard EvaluationResults.from_rowar_results against an empty rowar_results list, which would otherwise crash with an IndexError when the run is interrupted before any results are collected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #436
Problem
When an evaluation run is interrupted (e.g. via
KeyboardInterrupt/ Ctrl-C) orraises an unexpected exception,
write_evaluation_run_endwas never called.The run would remain permanently shown as Running in ell-studio, and no new
runs could be added to the same logdir without creating a fresh one.
The
TODOcomment in the original code acknowledged this was missing:Solution
BaseExceptionhandler (catches both regular exceptions andKeyboardInterrupt) that setssuccess=False, records the error string,and calls
write_evaluation_run_endbefore re-raising. This ensures the runis always finalized in the store, regardless of how execution exits.
EvaluationResults.from_rowar_resultsagainst an emptyrowar_resultslist, which would previously raise an
IndexErrorwhen the run is interruptedbefore any results are collected.
Testing
Manually verified by running an evaluation and pressing Ctrl-C mid-run: